Mobile device utilizing multiple cameras

ABSTRACT

A mobile electronic device may have two rear facing cameras to capture environmental information. A camera and a depth sensor of the mobile electronic device may capture biometric facial data for security verification. Characteristics of the environment may be identified based on the environmental information and results of the security verification.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.15/433,707, filed Feb. 15, 2017, which is a continuation of U.S. patentapplication Ser. No. 14/971,483, filed Dec. 16, 2015, which issued asU.S. Pat. No. 9,578,200 on Feb. 21, 2017, which is a continuation ofU.S. patent application Ser. No. 13/480,107, filed May 24, 2012, whichissued as U.S. Pat. No. 9,218,526 on Dec. 22, 2015, which isincorporated by reference as if fully set forth.

TECHNICAL FIELD

This application is related to an apparatus and method to detect a paperdocument, book, or other objects using one or more sensors.

BACKGROUND

Electronic devices such as tablets, mobile devices, cellular phones,personal digital assistants, smartphones, tablet personal computers(PCs), laptop computers, notebook computers, televisions, digitalpicture frames, large displays, or the like, have helped to reduce thevolume of printing and increase paperless environments. Nevertheless,printed content still plays a significant portion of life and business.Even with the many benefits of digital content, printed content is stilldesirable.

More and more electronic devices, especially mobile computers, are beingbuilt with cameras and other sensors to capture and gather information.However, using such instruments for accurate, quick, and seamlessdetection of papers, documents, and other objects still is lacking.

Current work environments look to digitize information to do work on acomputer and manage documents. This brings the real-world into thedigital world where information and documents mostly flow one directioninto the digital world. However, bringing the digital world to areal-world work environment is also desired since many documents areprinted and people enjoy reading books. In this working model,information mostly flows from the digital world to the real-world.

Therefore, it is desirable to detect paper documents, books, and otherobjects using one or more image sensors to provide an enhancedpaper-digital environment.

SUMMARY

An apparatus and method for detecting paper documents, books, or otherobjects using one or more sensors, such as cameras, on one or morecomputing devices is disclosed. The one or more computing devicescommunicate and share information such that a paper document, book, orother object in a given space is detected and identified. Onceindentified, information relevant to the paper documents, books, orother objects is retrieved.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding may be had from the following description,given by way of example in conjunction with the accompanying drawingswherein:

FIG. 1 is a diagram of an electronic device;

FIG. 2 is a diagram showing devices for detecting at least one object,book, or paper document using one or more sensors;

FIG. 2a is a diagram of gestures or inputs;

FIG. 3 is another diagram of an apparatus to detect at least one object,book, or paper document using one or more sensors;

FIG. 4 is a process for identifying at least one object, book, or paperdocument; and

FIG. 5 is an interactive process related to a book or paper document.

DETAILED DESCRIPTION

Examples will be described with reference to the drawing figures whereinlike numerals represent like elements throughout. For the methods andprocesses described below the steps recited may be performed out ofsequence in any order and sub-steps not explicitly described or shownmay be performed. In addition, “coupled” or “operatively coupled” maymean that objects are linked between zero or more intermediate objects.Also, any combination of the disclosed features/elements may be used inone or more embodiments. When using referring to “A or B”, it mayinclude A, B, or A and B, which may be extended similarly to longerlists.

U.S. Patent Publication No. 2011-0199342 is herein incorporated byreference as if fully set forth and may be used in combination with thegiven examples to provide a display device that may adaptively emitultrasound, ultrasonic, acoustic, or radio waves to provide an elevated,indented, or texturized sensation to an object, person, or paperdocument near a display device.

FIG. 1 is a diagram of an object device 100 that may be a wirelesssubscriber unit, mobile device, user equipment (UE), mobile station,smartphone, pager, mobile computer, cellular phone, cellular telephone,telephone, personal digital assistant (PDA), computing device, surfacecomputer, tablet, tablet computer, monitor, general display, versatiledevice, digital picture frame, appliance, automobile computer system,television device, home appliance, home computer system, laptop,netbook, personal computer (PC), an Internet pad, digital music player,peripheral, add-on, an attachment, virtual reality glasses, mediaplayer, video game device, head-mounted display (HMD), helmet mounteddisplay (HMD), glasses, goggles, a wearable computer, wearable headsetcomputer, or any electronic device for mobile or fixed applications.

Object device 100 comprises computer bus 140 that couples one or moreprocessors 102, one or more interface controllers 104, memory 106 havingsoftware 108, storage device 110, power source 112, and/or one or moredisplays controller 120. In addition, object device 100 comprises anelevation, indenting, or texturizing controller 121 to providesensations to an object or person located near one or more displaydevices 122.

One or more display devices 122 can be configured as a plasma, liquidcrystal display (LCD), light emitting diode (LED), field emissiondisplay (FED), surface-conduction electron-emitter display (SED),organic light emitting diode (OLED), flexible OLED, or any other displaydevice. The one or more display devices 122 may be configured,manufactured, produced, or assembled based on the descriptions providedin U.S. Patent Publication Nos. 2007-247422, 2007-139391, 2007-085838,or 2006-096392 or U.S. Pat. No. 7,050,835 or WO Publication No.2007-012899 all herein incorporated by reference as if fully set forth.In the case of a flexible or bendable display device, the one or moreelectronic display devices 122 may be configured and assembled usingorganic light emitting diodes (OLED), liquid crystal displays usingflexible substrate technology, flexible transistors, field emissiondisplays (FED) using flexible substrate technology, or the like.

One or more display devices 122 can be configured as a touch ormulti-touch screen display using resistive, capacitive, surface-acousticwave (SAW) capacitive, infrared, strain gauge, optical imaging,dispersive signal technology, acoustic pulse recognition, frustratedtotal internal reflection, or magneto-strictive technology, asunderstood by one of ordinary skill in the art. One or more displaydevices 122 can also be configured as a three dimensional (3D),electronic paper (e-paper), or electronic ink (e-ink) display device.

Coupled to one or more display devices 122 may be pressure sensors 123.Coupled to computer bus 140 are one or more input/output (I/O)controller 116, I/O devices 118, global navigation satellite system(GNSS) device 114, one or more network adapters 128, and/or one or moreantennas 130. Examples of I/O devices include a speaker, microphone,keyboard, keypad, touchpad, display, touchscreen, wireless gesturedevice, a digital camera, a digital video recorder, a vibration device,universal serial bus (USB) connection, a USB device, or the like. Anexample of GNSS is the Global Positioning System (GPS).

Object device 100 may have one or more motion, proximity, light,optical, chemical, environmental, barometric, atmospheric pressure,moisture, acoustic, heat, temperature, radio frequency identification(RFID), biometric, face recognition, image, photo, or voice recognitionsensors 126 and touch detectors 124 for detecting any touch inputs,multi-touch inputs, or multiple touch inputs for one or more displaydevices 122. Sensors 126 may also be an accelerometer, an electroniccompass (e-compass), gyroscope, a 3D gyroscope, 3D accelerometer, or thelike. One or more interface controllers 104 may communicate with touchdetectors 124 and I/O controller 116 for determining user inputs toobject device 100. Coupled to one or more display devices 122 may bepressure sensors 123 for detecting presses on one or more displaydevices 122.

Ultrasound source/detector 125 may be configured in combination withtouch detectors 124, elevation, indenting, or texturizing controller121, one or more display devices 122, pressure sensors 123, or sensors126 to project or generate ultrasound waves, rays, or beams to an objectto simulate elevated, indented, or texturized sensations, recognizeinputs, or track the object. There may be cases for input recognition orobject tracking wherein an ultrasound is provided without detectedsensation to the object.

Still referring to object device 100, storage device 110 may be any diskbased or solid state memory device for storing data. Power source 112may be a plug-in, battery, solar panels for receiving and storing solarenergy, or a device for receiving and storing wireless power asdescribed in U.S. Pat. No. 7,027,311 herein incorporated by reference asif fully set forth.

One or more network adapters 128 may be configured as a FrequencyDivision Multiple Access (FDMA), single carrier FDMA (SC-FDMA), TimeDivision Multiple Access (TDMA), Code Division Multiple Access (CDMA),Orthogonal Frequency-Division Multiplexing (OFDM), OrthogonalFrequency-Division Multiple Access (OFDMA), Global System for Mobile(GSM) communications, Interim Standard 95 (IS-95), IS-856, Enhanced Datarates for GSM Evolution (EDGE), General Packet Radio Service (GPRS),Universal Mobile Telecommunications System (UMTS), cdma2000, widebandCDMA (W-CDMA), High-Speed Downlink Packet Access (HSDPA), High-SpeedUplink Packet Access (HSUPA), High-Speed Packet Access (HSPA), EvolvedHSPA (HSPA+), Long Term Evolution (LTE), LTE Advanced (LTE-A), 802.11x,Wi-Fi, Zigbee, Ultra-WideBand (UWB), 802.16x, 802.15, Wi-Max, mobileWi-Max, Bluetooth, radio frequency identification (RFID), Infrared DataAssociation (IrDA), near-field communications (NFC), or any otherwireless or wired transceiver for communication via one or more antennas130. One or more network adapters 128 may also be configured forautomobile to automobile, car to car, vehicle to vehicle (V2V), orwireless access for vehicular environments (WAVE) communication. One ormore network adapters 128 may also be configured for human bodycommunications where the human body is used to communicate data betweenat least two computers coupled to the human body. In addition, any ofthe communication links referenced herewith may be wired or wireless orboth wired and wireless.

Any of devices, controllers, displays, components, etc. in object device100 may be combined, made integral, or separated as desired. Forinstance, elevation, indenting, or texturizing controller 121 may becombined with ultrasound source/detector 125 in one unit. Moreover, thedescriptions below reference the detection and identification ofobjects, books, or paper documents. However, the apparatuses and methodsdescribed herein may also apply to providing information, detecting,recognizing, or identifying magazines, periodicals, journals, whitepapers, brochures, folders, or like.

FIG. 2 is a diagram showing devices for detecting at least one object216, book 215, or at least one of paper documents 212-214 in workspace202 using one or more sensors. Although many of the examples forthcomingwill be for detecting book 215 or at least one of paper documents212-214, the same descriptions may apply to at least one object 216.Devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ may be configured as mobile orfixed mobile computing devices with at least some of the components andparts provided in object device 100. Devices 206 ₁, 208 ₁, 210 ₁, or 224₁ each may be provided with one or more sensors 206 ₂, 208 ₂, 210 ₂, or224 ₂ respectively, that may each be configured as a camera, a fieldcamera, a camera with a 180 degree fisheye lens, an infrared camera, aninfrared sensor, an infrared transceiver, a Microsoft Kinect motionsensing device, a Microsoft Kinect motion sensing system, a depthsensor, a laser finder, a stereo vision sensor, or a special papersensor. Sensor devices for detecting and tracking objects are given byU.S. Patent Publication Nos. 2010-0201811, 2011-0025827, 2011-0096182,2011-0188054, 2011-0188054, and 2012-0050488 all herein incorporated byreference as if fully set forth. In the examples given herewith, sensors206 ₂, 208 ₂, 210 ₂, or 224 ₂ may be used in combination with any ofsensors 126 for detecting at least one object 216, book 215, or at leastone of paper documents 212-214 in workspace 202. Although not shown, oneor more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂ may be on other sides,such as the back of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. For instance,many smartphones, tablets, netbooks, and laptops have front and rearcamera sensors.

Workspace 202 may be a certain room, space, virtual room, collaborativeenvironment, meeting room, or the like. Workspace 202 may be configuredwith special markers, dots, or other indicators to help or assist thedetection of at least one of paper documents 212-214 or book 215 by oneor more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂. The markers, dots, orother indicators may be visible or invisible to the human eye but stilldetectable by one or more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂.

One or more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂ may individually,collectively, partially, or in part detect a segment, portion, or pieceof content or information on at least one of paper documents 212-214 orbook 215 for identification recognition, or providing information.Detection may be performed on the front, back, or both the front andback portions of at least one of paper documents 212-214 or book 215.Using both front and back portions, detection may allow double-sideddetection to identify at least one of paper documents 212-214 or book215. Any detected content or information may be shared via communicationlinks 226 or 228 using one or more network adapters 128. Communicationlinks 226 or 228 may operate using Wi-Fi direct, Digital Living NetworkAlliance (DLNA), peer-to-peer communication, adhoc networking, or anyother device to device communication protocol.

In addition, one or more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂ may becoordinated to collectively detect edges, text, characters, images,photos, or any other information on at least one of paper documents212-214 or book 215 for identification. Optical character recognitionmay be used by any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ or oneor more servers 222 in connection with the detected data. Moreover, thecharacteristics, type, or category of paper documents 212-214 or book215 may be determined based on the detected dimensions or color of thepaper. For instance, some work environments use legal sized paper forcontracts or green paper for billing.

If a camera device is configured in one or more sensors 206 ₂, 208 ₂,210 ₂, or 224 ₂, a digital autofocus, digital autozoom, digital autopan,or adaptive aperture control operation may be performed by any one ofdevices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ to adaptively capture differentportions or segments of at least one of paper documents 212-214 or book215. This may allow user-friendly capture from various flexiblepositions by any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. Moreover,one or more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂ may be configured toallow adaptive mechanical movement or digital eye detection or trackingof at least one of paper documents 212-214 or book 215.

Any detected content or information may be shared via communicationlinks 209, 226, 228, or 221 using one or more network adapters 128 overnetwork 218 for identification, recognition, or providing information.With using multiple devices and multiple cameras, multicameras orsensors, different view points and angles of at least one of paperdocuments 212-214 or book 215 may be captured. This may create adocument environment or operation that is ubiquitous, seamless, easy, orinvisible to a user. It may also allow the user to normally use devices206 ₁, 208 ₁, 210 ₁, or 224 ₁ and at least one of paper documents212-214 or book 215 since the devices or at least one of paper documents212-214 or book 215 can be orientated differently for capture oridentification purposes.

The sharing or management of captured information by devices 206 ₁, 208₁, 210 ₁, or 224 ₁ may be performed by client based software or bysoftware on one or more servers 222 where the information iscommunicated over communication link 220. Client based software may bepart of the operating system found in memory 106. Client based softwaremay also be collaborative or work management software such as WebEx orGoToMeeting. The client or network based software may be part of adocument management system, workflow system, records management system,enterprise content management (ECM) system, digital asset managementsystem, or a web-based service. The client based software may also be athin client.

Detected content or information on the front, back, or both the frontand back portions of at least one of paper documents 212-214 or book 215may be one or more quick response (QR) codes, one or more watermarks,invisible markings, an international standard book number (ISBN), aninvisible code, information printed using a special ink, an invisiblebar code, a unique tracking number, a document management number, aunique marking, or one or more printed labels. Any of the identifyinginformation may also be on the top, bottom, or both top and bottomportions of at least one of paper documents 212-214 or book 215. Anexample of a printing or smart printing apparatus providing specialmarkings that may uniquely identify each sheet of printed content, suchas by pointers, is given in U.S. Pat. No. 6,964,374 herein incorporatedby reference as if fully set forth.

With at least one unique code or identifier for a specific physical copyor print-out, different hard copies or versions of at least one of paperdocuments 212-214 may be tracked for security, copyright management,digital rights management, rights management, or document managementpurposes. The unique code or identifier may also be detected by a smartshredder (not shown) and shared with any one of devices 206 ₁, 208 ₁,210 ₁, or 224 ₁ or servers 222 to record when at least one paperdocuments 212-214 was destroyed.

Identification may be performed partially, collaboratively, orcompletely by any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ byassembling, collecting, processing, or rendering the detected datacaptured by each device. For paper document 214, device 208 ₁ using oneor more sensors 208 ₂ may detect certain portions of text, image, orother data on paper document 214 from a first angle. For paper document214, device 210 ₁ using one or more sensors 210 ₂ may detect othercertain portions of text, image, or other data on paper document 214from a second angle. Devices 208 ₁ and 210 ₁ may then share, viacommunication link 228 or any other network, the detected portionscaptured from the different angles and query a database in the devicesor on one or servers 222 to identify paper document 214. This operationmay similarly be scaled to devices 206 ₁ and 224 ₁ to quickly andefficiently identify paper document 214. It may similarly be used toidentify paper documents 212 and 213, book 215, or object 216.

Identification may also be performed partially or completely by one ormore servers 222. Image, photo, text, or character recognition operationmay be performed to identify at least one of paper documents 212-214 orbook 215. Examples of image, photo, text, or character recognitionengines are provided by U.S. Patent Publication Nos. 2011-0110594 or2012-0102552 all herein incorporated by reference as if fully set forth.In addition, identification may be performed by a document mosaicingalgorithm performed partially or completely by any one of devices 206 ₁,208 ₁, 210 ₁, or 224 ₁.

Once identified, a digital version or copy of any one of paper documents212-214 or book 215 may be retrieved by any one of devices 206 ₁, 208 ₁,210 ₁, or 224 ₁. The digital version or copy may be retrieved frommemory 106 or storage device 110 in at least one of devices 206 ₁, 208₁, 210 ₁, or 224 ₁ using network 218 or communication links. The digitalversion or copy may also be retrieved from one or more servers 222 usingcommunication link 220 over network 218. In addition, the digitalversion or copy may be retrieved by or from a cloud server configured onone or more servers 222 over communication link 220.

Moreover, any information such as the author, author profile, version,history, timestamps, location tracking information, position tracking,social networking related information, or cloud based information may beretrieved from memory 106 or storage device 110 in any one of devices206 ₁, 208 ₁, 210 ₁, or 224 ₁. Location tracking information mayindicate what offices or desks the at least one of paper documents212-214 has visited. This same information may also be retrieved fromone or more servers 222 over network 218. This information may be usedfor smart paper tracking, paper analytics, or real-world documentmanagement.

Prior to retrieval of a digital version or any information by any one ofdevices 206 ₁, 208 ₁, 210 ₁, or 224 ₁, a password or any other securityverification process may be initiated such that a print-out ofconfidential information that needs to be secure may still be securewith the digital copy. Security verification may include any biometric,voice, or sensor based check by at least one of sensors 126. Moreover,having a print-out of least one of paper documents 212-214 may indicateto any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ or servers 222 thatthe person in possession of the paper documents has permission toretrieve, view, or edit the digital copy. The digital copy may otherwisebe protected, locked, or encrypted.

In addition, once identified more printed or reprinted copies of atleast one of paper documents 212-214 may also be created of the same oraltered versions of the document providing a hybrid workspaceenvironment. Once identified, a timestamp for the time of identificationmay be stored in any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ orservers 222.

Once identified, the position of at least one of paper documents 212-214may be digitally tracked in workspace 202 by any one of devices 206 ₁,208 ₁, 210 ₁, or 224 ₁ with one or more sensors 206 ₂, 208 ₂, 210 ₂, or224 ₂. With tracking, at least one of paper documents 212-214 can bemoved from different screens and shared, emailed, altered, or the likeby any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. With tracking, atleast one of paper documents 212-214 can be located in the real-world ona desk for a certain time period, in an office, a particular building,or in a remote location. Book 215 may be used similarly in thisenvironment across multiple devices depending on digital rightsmanagement or copyrights.

Also, once identified, one or more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂may be configured to face or eye track a user using a facial recognitionor an eye tracking engine by any one of devices 206 ₁, 208 ₁, 210 ₁, or224 ₁ to pull up information when at least one of paper documents212-214 or book 215 is gazed, looked at, or read. Moreover, a timelineand history of at least one of paper documents 212-214 may be displayedon one or more display devices 122 on any one of devices 206 ₁, 208 ₁,210 ₁, or 224 ₁ in connection with recorded user activity in any giventime frame, order, etc. This way, a user's or worker's work habits maybe tracked and any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ may actas digital eyes to see at least one of paper documents 212-214 as a usersees it.

In addition, once at least one of paper documents 212-214 or book 215 isidentified, a mental snapshot of content that was being reviewed at acertain time may be determined or retrieved by any one of devices 206 ₁,208 ₁, 210 ₁, or 224 ₁. The exact position within workspace 202 at agiven time of documents related to at least one of paper documents212-214 may also be provided by any one of devices 206 ₁, 208 ₁, 210 ₁,or 224 ₁ or servers 222. This may provide what digital and real-worlddocuments were being referenced or track citations while working on anyone of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. It may also provide whatinformation was used in the digital and the real-worlds while working onany one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. Any of these snapshotsof past work environments or desktops by any one of devices 206 ₁, 208₁, 210 ₁, or 224 ₁ may be stored in storage device 110 or remotely andrecreated in the future.

Tracking may also allow determination of when and in what physicalfolder at least one of paper documents 212-214 is filed in workspace202. Accordingly, the digital version of at least one of paper documents212-214 is subsequently filed in the same or similar folder online inany one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ or servers 222.

Once at least one of paper documents 212-214 or book 215 are identified,one or more sensors 206 ₂, 208 ₂, 210 ₂, or 224 ₂ may be configured todetect or track hand, finger, palm, or any other body gestures made onor to at least one of paper documents 212-214. Using components inobject device 100, configurations of devices 206 ₁, 208 ₁, 210 ₁, or 224₁ may use infrared sensors, infrared cameras, infrared transceivers,cameras, acoustic waves, or ultrasound transducers to detect or trackgestures. Tracking or gesture detection engines or systems are given byU.S. Patent Publication Nos. 2010-0201811, 2011-0025827, 2011-0096182,2011-0188054, 2011-0211044, 2011-0188054, 2012-0050488, U.S. Pat. No.8,166,421, and PCT Publication No. WO 12/011044 all herein incorporatedby reference as if fully set forth.

A tracking engine may determine if a user is reading a part, section,paragraph, sentence, word, or page of at least one of paper documents212-214 or book 215. Subsequently, any one of devices 206 ₁, 208 ₁, 210₁, or 224 ₁ may display information on one or more display devices 122in real-time relevant to the part, section, paragraph, sentence, word,or page to enhance reading and productivity. This may be done by any oneof devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ using 206 ₂, 208 ₂, 210 ₂, or224 ₂ based on tracking gestures or finger placement. Moreover, sectionnumbers, paragraph numbers, or page numbers may be detected to quicklydetermine which section of at least one of paper documents 212-214 isbeing currently read by a user.

Moreover, a part, section, paragraph, sentence, or page of at least oneof paper documents 212-214 or book 215 may be highlighted or edited by auser using various gestures, hand motions, finger motions, or bodymotions. FIG. 2a is a diagram of gestures or inputs where a part,section, paragraph, sentence, word, or page of at least one of paperdocuments 212-214 or book 215 may be highlighted or edited by a circlinggesture 232, underlining gesture 234, underlining gesture 236, making atwo-handed “L-shaped” box gesture 238, crossing-out gesture 240, orcrossing-out gesture 242 detected by one or more sensors 206 ₂, 208 ₂,210 ₂, or 224 ₂. Any changes made may be soon reflected on a projectionscreen proximate to workspace 202 with projector device 132. In additionto body gestures, voice or speech commands may be detected by any one ofdevices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ to be used alone or in combinationwith gestures. For instance, tracking may be commenced via a voicecommand.

Around workspace 202, while the user is making edits, manipulating,cleaning up, writing hand-written notes, or writing virtual notes on atleast one of paper documents 212-214 or book 215, the digital versionretrieved by any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ may alsobe updated or digitally corrected in real-time.

In addition, once at least one of paper documents 212-214 or book 215are identified one or more sensors 126, 206 ₂, 208 ₂, 210 ₂, or 224 ₂may be configured to detect when content is grabbed or selected on atleast one of paper documents 212-214 or book 215 and thrown or placed onone or more display devices 122 on any one of devices 206 ₁, 208 ₁, 210₁, or 224 ₁. The content may be any one of text, paragraphs, sections,characters, images, photos, pages, or the like. For example, informationon at least one of paper documents 212-214 or book 215 may be selectedto fill in fields of a form displayed on one or more display devices 122of any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. If the informationon at least one of paper documents 212-214 or book 215 is an image oftext, it may be converted automatically to text or characters via anoptical character recognition engine once placed on one or more displaydevices 122 on any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. Thismay be helpful in filling out digital forms from hand-written paperforms or notes.

Moreover, a pinch or grab gesture may be used to select content on atleast one of paper documents 212-214 or book 215. After selection, athrow, tossing, or any other gesture motion may be used to digitallyplace the selected content on one or more display devices 122 on any oneof devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁.

In addition, after any one of the selection gestures given above, auser's hand, finger, arm, or any other body part may be tracked by oneor more sensors 126, 206 ₂, 208 ₂, 210 ₂, or 224 ₂ until a touch ormulti-touch input or gesture is detected on one or more display devices122 by touch detectors 124 on any one of devices 206 ₁, 208 ₁, 210 ₁, or224 ₁ to determine where to digitally place, drop, or position theselected content. Once the touch input is detected, any one of devices206 ₁, 208 ₁, 210 ₁, or 224 ₁ retrieves and displays the selectedcontent from the digital version at the drop point or position. Theselected content on at least one of paper documents 212-214 or book 215may be placed on a digital desktop, in an application, in browser, on aphoto, or any other digital content on one or more display devices 122on any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. In addition totouch inputs, any of the input devices or methods described for objectdevice 100 above may be used to determine where to place, drop, orposition the selected content on one or more display devices 122 on anyone of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁. These features may be usedfor productivity, data entry, forms, gaming, word processing,spreadsheets, or the like.

While a user's hand, finger, or arm is tracked, ultrasoundsource/detector 125 may be configured in combination with touchdetectors 124, elevation, indenting, or texturizing controller 121, oneor more display devices 122, pressure sensors 123, or sensors 126 toproject or generate ultrasound waves, rays, or beams to the user's hand,finger, or arm. The ultrasound over air may simulate elevated, indented,or texturized sensations. The ultrasonic sensation may be provided untila touch or multi-touch input is detected on one or more display devices122 by touch detectors 124 on any one of devices 206 ₁, 208 ₁, 210 ₁, or224 ₁.

In the given examples of office work models, at least one of paperdocuments 212-214 or book 215 may be the focus instead of devices 206 ₁,208 ₁, 210 ₁, or 224 ₁ creating a smart paper centric environment orpaper centric computing environment. Moreover, the online digital worldis better linked or hyperlinked with the offline world of paperdocuments. The linking may be viewed as real-world hyperlinking. Thetracking of documents within a workspace, office, or building may createan accounting system for paper documents similar to digital documents.The tracking may also create a recorded lifecycle or timeline for thepaper document or paper and digital version of the document. Fromanother viewpoint, an apparatus and process is provided to help tocreate a smart or interactive paper environment.

Still referring to FIG. 2, part of workspace 202 may include a surfaceor tabletop computing portion 211 ₁ having one or more sensors 211 ₂ fordetecting or tracking at least one of paper documents 212-214 or book215. One or more sensors 211 ₂ may each be configured as a camera, afield camera, a camera with a 180 degree fisheye lens, an infraredcamera, an infrared sensor, an infrared transceiver, a Microsoft Kinectmotion sensing device, a Microsoft Kinect motion sensing system, a depthsensor, a laser finder, a stereo vision sensor, or a special papersensor. Surface or tabletop computing portion 211 ₁ may be configuredwith at least some of the components and parts provided in object device100.

Surface or tabletop computing portion 211 ₁ may provide another anglefor collecting information on at least one of paper documents 212-214 orbook 215 for identification, recognition, tracking, or providinginformation. Information captured by surface or tabletop computingportion 211 ₁ may be shared directly with device 210 ₁ via communicationlink 232 using one or more network adapters 128 for identification,recognition, or providing information. Communication link 232 mayoperate using Wi-Fi direct, Digital Living Network Alliance (DLNA),peer-to-peer communication, adhoc networking, or any other device todevice communication protocol. Any detected content or information mayalso be shared with any one of devices 206 ₁, 208 ₁, 210 ₁, or 224 ₁ viacommunication link 230 using one or more network adapters 128 overnetwork 218 to identify or track at least one of paper documents 212-214or book 215.

FIG. 3 is another diagram of an apparatus to detect at least one object,book, or paper document using one or more sensors. Integrated desktopcomputing device 302, on surface 300, may be configured with sensors 308₁-308 ₃. Integrated desktop computing device 302 may be configured withat least some of the components and parts provided in object device 100.Sensors 308 ₁-308 ₃ may each be configured as a camera, a field camera,a camera with a 180 degree fisheye lens, an infrared camera, an infraredsensor, an infrared transceiver, a Microsoft Kinect motion sensingdevice, a Microsoft Kinect motion sensing system, a depth sensor, alaser finder, a stereo vision sensor, or a special paper sensor.

At least one of sensors 308 ₁-308 ₃ may be used to detect paper document310 or book 313 by looking for markers or identifiers on the backportion or side 311 or 315, respectively. Markers or identifiers mayalso be searched for on front portion or side 312. The detectionalgorithm or engine may be based on the descriptions provided previouslyabove. Also, in the examples given herewith at least one of sensors 308₁-308 ₃ may be used in combination with any of sensors 126 for detectingpaper document 310 or book 313.

Integrated desktop computing device 302 may identify paper document 310or book 313 based on the descriptions provided previously above. Onceidentified, information related to paper document 310 or book 313 may beretrieved over a network from a server or another device (not shown) aspreviously explained above. With integrated desktop computing device302, a user may be situated in front and review paper document 310 orbook 313 while integrated desktop computing device 302 ubiquitouslybrings up or displays relevant information 304 and 306.

Device 3161 using one or more sensors 3162 may share information withintegrated desktop computing device 302 over communication link 314using one or more network adapters 128 to provide information, detect,recognize, or identify paper document 310 or book 313. Communicationlink 314 may also be used to share information once paper document 310or book 313 is identified. Communication link 314 may operate usingWi-Fi direct, Digital Living Network Alliance (DLNA), peer-to-peercommunication, adhoc networking, or any other device to devicecommunication protocol.

FIG. 4 is a process 400 for identifying at least one object, book, orpaper document in connection with some of the descriptions given above.A portion or segment of the at least one object, book, or paper documentis detected or captured by a device or multiple devices using sensors,such as a camera, infrared sensor, or a Microsoft Kinect like sensorsystem (402). The device or multiple devices may be computing devices orany of the devices previously provided in the descriptions above.Detected information is gathered or shared for the at least one object,book, or paper document between detection devices (404). The at leastone object, book, or paper document is identified by one or moredetection devices in connection with the gathered or shared information(406). Information related to the at least one object, book, or paperdocument is retrieved or shared by one or more devices (408).

FIG. 5 is an interactive process 500 related to a book or paper documentin connection with some of the descriptions given above. A paperdocument or book is identified by at least one device based on one ofthe processes provided in the description above (502). A gesture isdetected by the at least one device, as explained above, on at least oneportion of the paper document or book using one or more sensors (504).The body part motion is subsequently tracked by one or more sensors(506). A touch input or input on a display device of the at least onedevice is detected (508). Information from the at least one portion ofthe paper document or book at the touch input location is digitallyretrieved and placed or displayed on the display of the device (510).

Although features and elements are described above in particularcombinations, each feature or element may be used alone without theother features and elements or in various combinations with or withoutother features and elements. The methods, processes, or flow chartsprovided herein may be implemented in a computer program, software, orfirmware incorporated in a computer-readable storage medium forexecution by a general purpose computer or a processor. Examples ofcomputer-readable storage mediums include a read only memory (ROM), arandom access memory (RAM), a register, cache memory, semiconductormemory devices, magnetic media such as internal hard disks and removabledisks, magneto-optical media, and optical media such as CD-ROM disks,digital versatile disks (DVDs), and BluRay discs.

Suitable processors include, by way of example, a general purposeprocessor, a multicore processor, a special purpose processor, aconventional processor, a digital signal processor (DSP), a plurality ofmicroprocessors, one or more microprocessors in association with a DSPcore, a controller, a microcontroller, Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, anyother type of integrated circuit (IC), and/or a state machine.

A processor in association with software may be used to implementhardware functions for use in a computer, or any host computer. Theprogrammed hardware functions may be used in conjunction with modules,implemented in hardware and/or software, such as a camera, a videocamera module, a videophone, a speakerphone, a vibration device, aspeaker, a microphone, a television transceiver, a hands free headset, akeyboard, a Bluetooth® module, a frequency modulated (FM) radio unit, aliquid crystal display (LCD) display unit, an organic light-emittingdiode (OLED) display unit, a digital music player, a media player, avideo game player module, an Internet browser, and/or any wireless localarea network (WLAN) or Ultra Wide Band (UWB) module.

Any of the displays, processors, memories, devices or any othercomponent disclosed may be configured, produced, or engineered usingnanotechology based nanoparticles or nanodevices.

What is claimed is:
 1. A mobile electronic device comprising: at leasttwo rear facing cameras configured to capture a part of an environmentand to capture a part of a paper document in the environment; a cameraand a depth sensor configured to capture biometric facial informationfor security verification; a processor configured to identifycharacteristics of the environment based on the captured part of theenvironment, the captured part of the paper document, and informationreceived via a network adapter, wherein the identification is performedsubsequent to a successful result of the security verification; theprocess configured to detect, based on the at least two rear facingcameras, a user gesture on the paper document, wherein the user gestureis a command to digitally edit or highlight a portion of the paperdocument; and the processor configured to retrieve additionalinformation of the identified characteristics of the environment.
 2. Themobile electronic device of claim 1 further comprising: the processor isconfigured to utilize assistance information for the identification; andwherein the assistance information is obtained, by the mobile electronicdevice, from the captured part of the environment.
 3. The mobileelectronic device of claim 1, wherein eye tracking is utilized to trackposition or location of the mobile electronic device.
 4. The mobileelectronic device of claim 1, wherein a timeline or history of theidentified characteristics of the environment is retrieved.
 5. A methodperformed by a mobile electronic device, the method comprising:capturing, by at least two rear facing cameras of the mobile electronicdevice, a part of an environment and a part of a paper document in theenvironment; capturing, by a camera and a depth sensor of the mobileelectronic device, biometric facial information for securityverification; identifying, by a processor of the mobile electronicdevice, characteristics of the environment based on the captured part ofthe environment, the captured part of the paper document, andinformation received via a network adapter, wherein the identificationis performed subsequent to a successful result of the securityverification; detecting, based on the at least two rear facing camerasby the processor a user gesture on the paper document, wherein the usergesture is a command to digitally edit or highlight a portion of thepaper document; and retrieving additional information of the identifiedcharacteristics of the environment.
 6. The method of claim 5 furthercomprising: utilizing, by the processor, assistance information for theidentification; and obtaining, by the mobile electronic device, theassistance information from the captured part of the environment.
 7. Themethod of claim 5, wherein eye tracking is utilized to track position orlocation of the mobile electronic device.
 8. The method of claim 5,wherein a timeline or history of the identified characteristics of theenvironment is retrieved.